Lightfield cameras are able to measure the amount of light traveling along each ray bundle that intersects the image sensor by using a micro lens array that is placed between the main lens and the image sensor. This light field is then post-processed to reconstruct images of the scene from different points of view. The light field also permits a user to change the focal point of the images.
FIG. 1 shows illustrative configuration of a lightfield camera. The lightfield camera 1 comprises a lens arrangement associated with an image sensor array 13. The image sensor array 13 comprises a large number p of photosites 131, 132, 133 to 13p arranged in the form of a grid of X columns and Y lines, n being a number of elements corresponding to X times Y. A color filter array 12 is arranged on the image sensor array 13. The color filter array 12 typically arranges RGB (Red, Green and Blue) color filters on the image sensor array 13, the RGB arrangement can be, for example, the form of a Bayer filter mosaic. The lens arrangement comprises a primary lens 10, also called main lens, and a lenslet array 11 which comprises a plurality of m microlenses 111, 112, 11m, m being a positive integer. The microlenses 111, 112, 11m are arranged in such a way as to each be optically associated with a plurality of photosites 131, 132, 133 to 13p. The number of photosites 131, 132, 133 to 13p are optically associated with one microlens that corresponds to the number of views of the scene acquired with the lightfield camera. To obtain the different views, the raw image (i.e. the color sensor data acquired with the image sensor array 13) is demosaicked then de-multiplexed. After the demosaicking step, RGB image data values are obtained at each photosite location for each view.
The captured images of the scene with a lightfield camera should undergo view demultiplexing, i.e., the data conversion from the 2D raw image to the 4D light-field. The demultiplexing process consists of reorganizing the photosites of the raw image in such a way that all photosites 131 capturing the light rays with a certain angle of incidence are stored in the same image creating sub-aperture views. Each sub-aperture view is a projection of the scene under a different angle. The set of sub-aperture views create a block matrix where the central sub-aperture view stores the photosites capturing the light rays that pass through the central section of the main lens. In fact, the angular information of the light rays is provided by the relative photosites positions in the microlens images in respect to the microlens image centers.
One of the drawbacks in the use of lightfield cameras is the vignetting effects that cause darker views (due to the less luminance values) in peripheral sub-aperture views.
FIG. 1 illustrates such a drawback of a lightfield camera due to less refracted light energy in peripheral sub-aperture views. The dashed line represents light rays which have less refracted light energy due to vignetting effect.
Due to the optical imperfection of the main lens, higher incidence angles bring less refracted light energy. In a 2D mode, the result is a brightness non-uniformity of the image, where corners are darker than the center. In lightfield camera, peripheral sub-aperture views are unusable since they are too dark compared to central sub-aperture views. As illustrated in FIG. 1, peripheral sub-aperture view 102 (with the collection of peripheral positioning photosites 131 and 133 with respect to the center of microlens images) is not usable since it is too dark, mostly because view 102 is under exposed and therefore noisy. On the other hand central sub-aperture view 101 with the collection of central positioning photosites 132 that captured the light ray passing through the main lens center to the photosite 132 is usable since view 101 is better-exposed and less noisy.
FIG. 2 shows an enlargement view of image sensor array 13 depicted in FIG. 1. Central photosites 132 capture the light ray that has passed through the center of the main lens to central photosites 132 while peripheral photosites 131 and 133 capture light rays incoming with oblique angle compared with the central photosites 132, the luminance level of peripheral photosite 131 and 133 is less due to several reasons. Firstly, the light ray incoming with oblique angle has a longer way to travel to the image corner. Secondly, the pupil seen by the off-axis point is not round but elliptical and has a smaller area than the round pupil seen by the central photosites 132. Thirdly, while the light hits the image center at normal incidence, the light strikes the image corner at the angle b. The combined effect of all cosine factors is a cos4law (cosine forth law) luminance falloff towards the image corners.
FIG. 3 shows an image of white signal in the central part of a sensor from image array 13. In this image, the vignetting effect nearly follows a symmetric cos4law (cosine forth law) fall-off. For the photosites of the central microlenses, the vignetting effect can be formulated by a cos4law (cosine forth law) factor or a Gaussian fall-off function, for example.
Although the fall off can be simulated with the cos 4 law or any other fall off calculations, those fall offs are merely theoretical fall off calculations. Therefore, a calibration step may be used to more accurately estimate the light fall off in peripheral photosites.
Also, apart from the signal amplitude problems, these vignetting effects lead also to color artifacts when applying content-aware demosaicking methods.
According to a referenced document, 20 Mar. 2015, N. Sabater, M. Seifi, V. Drazic, G. Sandri, and P. Perez, “Accurate Disparity Estimation for Plenoptic Images,” http://link.springer.com/article/10.1007%2F978-3-319-16181-5_42/lookinside/000.png the difference of the luminance level (between the center and the periphery) may be reduced using a weighting matrix derived by dividing the raw data by a corresponding white image (i.e., the image of a Lambertian light-emitting type object).
In above referenced document, it is explained that considering the white image as a weighting matrix is computationally more efficient than mathematically modeling the vignetting on every microlens. Mathematically modeling the vignetting effect on every microlens is impractical due to the lack of the precise knowledge of the camera's intrinsic parameters or about the design of each lens. Some problems with the proposed solution of the referenced document is that a Lambertian light-emitting type object image needs to be captured in order to calculate the weighting matrix whenever the camera parameters (zoom/focus) are changed because the position of the microlens images on the photosites depends on the camera parameters, e.g., zoom and focus of the camera. That is, capturing Lambertian light-emitting type object images for the purposes of calculating the weighting matrix whenever the camera parameters (zoom/focus) are changed is impractical.
Additionally, reference document EP15306059.5 discloses a way to solve vignetting problem by introducing a gradient filter between the primary lens and the photosites set at the location of the stop aperture. The drawback of this approach is that it still needs Lambertian light-emitting type object image to be captured whenever changing the focus/zoom of the camera in order to estimate the position of the microlens images on the photosites.